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This paper proposes that instructional validity (i.e., the extent to which 
teachers actually teach skills tested) should be part of the test development 
process. The development of a districtwide ninth grade math exam is used to 
illustrate how a study of the instructional process conducted during the early 
phases of test development can benefit the curriculum and strengthen the match 
between instruction and assessment. Data collected from teachers 9 instructional 
records were analyzed in light of Rasch difficulty and fit statistics for each 
skill. Additional insights based upon teachers' responses on an en3-of-year 
questionnaire were discussed. 



INTRODUCTION 

Ensuring, a match 'between test items and curriculum content is a basic re- 
quirement for development of criterion- referenced tests. Test specifications, 
experts' Judgmental reviews of test items, and item analysis procedures are com- 
monly employed during the stages of test development to ^investigate and verify 
a test f s content validity (the extent to which test content appropriately 
assesses specific skills of a particular curriculum). The courts have ruled, 
however, that demonstration of a test's content validity is not a sufficient 
condition for establishing that a test is "fundamentally fair in form and prac- 
tice" ( Debra P. v. Turlington « 1981). When tests sure used for purposes which 
may deny constitutional guarantees of equal protection or due process — for 
example, when they are used to deny high school diplomas or grade promotion — 
the courts require that test users provide evidence that the test assesses 
skills actually taught in the classroom* School authorities cannon assume that 
state- or locally-mandated curricular objectives are taught uniformly in all 
classrooms. Rather, they must clearly demonstrate a test's "instructional 
validity" or the degree to which a test measures what has been taught to 
examinees. 

Most attempts to establish that a test is instructionally valid for a 

particular curriculum have been post hoc . That is, the extent to which students 
• r^™^~~* «^^^— 

received instruction in skills assessed by a test has been examine after :he 
test has been developed and administered. A prime example is the SSAT-II, 
an ex ami nation required for high school graduation in the state of Florida and 
the subject of the Debra P. case. , When use of this test was challenged, the 
court directed the state to produce evidence of its instructional validity. In 
situations where validity studies of a -post hoc nature are performed, test users 
must resolve discrepancies between instructional content and test content by 
either revising their list of skills to match the test, or revising the test to 



mat: . jTiction, and/or mandating and enforcing instruction of skills tested. 
He \ . .ese alternatives is particularly attractive once the test has been 

Hationc r the Study 

/ Stuu ng the congruence between test items and classroom instruction 
prior to the finalization of a test has certain advantages. First, problems 
of instructional validity can be avoided if data on the adequacy of instruction 
in skill* assessed by test items are collected concurrently with the piloting 
of those items. Simultaneous collection of instructional data and pilot test 
information could result in saving time, energy, dollars and, some cases, 
litigation and postponement of the testing program. When a poor match is dis- 
covered in the early phases of test development, two options are available. 
Either objectives can be eliminated and matching test items deleted, or items 
can be retained and instructional staff can investigate reasons why teachers 
failed to teach the objectives in order to provide them with necessary assis- 
tance. 

A second advantage of understanding the relationship between test items 
and instruction related to the items deals with the increased power that 
such an understanding would give to pyschometric analyses. Many new procedures 
for analyzing test items have been advanced in recent years. The rationale 
for these procedures has been based primarily in technical consideration, and 
few if any of these procedures have been advanced as providing new insights 
into the process of instruction. If a technical procedure is proposed as 
useful for educational measurement, then the procedure should show seme 
sensitivity to known variation in instructional activities; The interpretation 
of psychometric analyses can be. greatly enriched. if the relationship between 
psychometric and instructional attributes of test items can be established. 



A third value in examining instructional information as part of the 
test development process is that it may help to re-establish teachers as part 
of the assessment process. Many teachers and other educators have failing 
faith in testing programs because they see such programs as separate and 
distinct from the instructional program* Educators involved in curriculum 
and instruction may become more involved and supportive of testing if the 
testing, co mmun ity begins to recognize the value of instructional information 
in developing test design to assess the effects of instruction. 

The testing program. Charleston County School District has integrated 
test development with an evaluation of locally-developed curriculum guides 
in elementary and middle school math and language arts and selected high 
school courses* In this paper the development of a districtvide subject area 
final exam for ninth grade General Mathematics I (G2H) will he used to illus- 
trate how a study of instructional validity can merge and henefit hoth cur- 
riculum and test development efforts* Data collected from teachers' instruc- 
tional records of GMI objectives were analyzed in light of (a) the percentage 
of items per objective classified as acceptable according to Rasch item fit 
statistics and (b) the Rasch average difficulty of acceptable items assessing 
the.. objective. It was hypothesized that (l) there would be a positive rela- 
tionship between the extent to which an objective was taught by teachers and 
the percentage of fitting items remaining after the analysis. It was further 
hypothesized that (2) there would be a negative relationship. between teachers' 
perceptions of the extent of students' mastery of an objective with the 
average difficulty of acceptable test items. (This hypothesis implies a 
postive relationship between students' mastery and item easiness.) Examples 
of how the results of the analyses assisted in revisions of the GMI curriculum 
and test are presented. 



METHOD 

GMI 

The GMI curriculum is partitioned into four units corresponding to the 
four nine-weeks grading periods* Each unit has from l8 to 31 objectives which 
are tested* School district staff, ninth grade math teachers and external 
consultants prepared test specifications and items for each unit. Items were 
piloted at the end of each quarter during the 1982-83 school year. The Basch 
model was used to generate item difficulties and fit statistics and to link 
the four sets of test forms. Fitting items were defined-^as those which had ." 
between-group fit statistics less than or equal to 3.0. A description of the 
technical issues regarding item analysis can be found in a report prepared by 
Educational Measurement Systems, Inc. (Ryan 4 Potter, 1983). The appendix 
contains the objectives covered by- the GMI test. 
Instructional Variables 

Airasian and Madaus (1983) describe various procedures which have been 
applied to associate testing with instruction. Some compare test performance 
to elements of instructional materials, such as textbook and syllabi. Others 
analyze data on classroom instruction, either through direct observation, 
teacher interviews or teacher logs. (See Leinhardt (l98l) for a review of 
these methods and for descriptions of studies in which they were used,) 

Teacher self- report of instruction and student mastery, the method used 
here, was one component of an evaluation of the course curriculum guide that 
vas conducted concurrently with the development of the GMI test. Teachers' 
estimates of students 1 opportunity to learn and their evaluations of students' 
level of mastery were gathered for GMI objectives. Fifty-one teachers were 
asked to assign one of the following descriptions of instruction and mastery 
to each objective as it applied to all students in their classes: 



1. This objective was taught and mastered by most students (T/M). 

2. Instruction was not necessary^ Most students have mastered this 
objective (NT/M). 

3. This objective was taught, but most students did not master it (T/NM) 
k. Insufficient instructional time; objective not taught (NT/ IT) . 

5. Objective not taught for some reason other than insufficient 
time (NT/0). 

Teachers who failed to enter an instructional code for an objective were coded 
as M No Response" (NR). 

Data from teachers 1 instructional- records were collected at the end of 
each quarter for objectives for that quarter and for previous quarters. At 
the end of the school year, dajba from. the four records were aggregated for 
each teachers and the percentage of teachers selecting each instructional code 
was calculated for each objective. Thirty-eight teachers (75^) returned com- 
pleted records for the first and second quarter; 3 1 * teachers {66%) returned 
forms at the end, of the third quarter; and 32 teachers (63%) returned them at 
the end of the year* The total number of enrolled students per teacher ranged 
from 16 to IkO with a mean of 55.1 and a standard deviation of 29.3. 
Analyses 

Pearson-Product Moment correlations were generated for the two test des- 
criptors — item difficulty and the percentage of fitting items per objective — 
and the percentage of teachers^ selecting each instructional code for each ob- 
jective. In addition, instructional codes were collapsed in order to obtain 
and correlate the following instructional variables: 

1. Mastered by students (M - T/M + NT/M). 
' 2. Not mastered by students (NM ■ T/NM + NT/ IT + NT/0). 

3. Taught by teachers (T = T/M +..T/NM). 
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J*. Not taught by teachers (NT = NT/M + NT/ IT +. NT/0). 
Initially, it was assumed that teachers who did not respond for a particular 
objective did not teach the objective. Therefore, teachers who did not re- 
spond .were added, in with NM and NT to create two additional variables: not 
mastered by most students plus no response (NMNO) and not taught by teachers 
plus no response - (lITNO). 

RESULTS 

General. Data Description 

Tables. 1- A through 1-D contain raw data for individual objectives from 
Tests 1 through U, respectively, used to derive the correlation coefficients: 
the percentage of teachers selecting each code on the GMI instruction record 
(T/M, NT/M, T/NM, NT/ IT, NT/0, BR); collapsed options (M, NM, NMNO, T, NT, ■ 
NTNO); average difficulty estimates for retained items; and the percentage of 
items fitting the Rasch model which were retained in the item pool. 

- Table 2 presents descriptive statistics — mean, standard deviation, 
range (minimum and. maximum) — for each variable. Overall, the average percen- 
tage of teachers teaching an objective was 60%. (Tests l-M , although the range 
(18-89) and standard deviation (20.6) indicate much variability in the degree 
to which different objectives were taught. The average percentage of teachers 
reporting that most of their students mastered a particular objective (5^.3) 
was similar to the mean for non-mastery (U2.8), as were the standard deviations 
and ranges. Again, much variability in student mastery of course objectives 
characterized GMI. 
Comparisons among Tests 

Comparisons among the four tests show significant .differences (Tests 1 
and 2 v. Teat-. 3 v~ Test h) . which have implications for the interpretation of 
the correlation coefficients. During the first two quarters the average 



percentage of teachers teaching a particular objectives was approximately jh%; 
the mean for mastery vas greater than 50%. Tests 3 and k show a trend for 
fewer objectives taught and a lower degree of student mastery. In addition, 
an average of 28.5% of the teachers did not enter instructional codes for 
Test h objectives. Though other reasons can be speculated, the two which were 
considered at the t im e were (a) the objectives -were not taught and teachers 
did not want central staff to know this or (b) teachers were under pressure 
at the end of the school year and had too much paperwork to complete their 
records. 

Statistics on the variables related to the GMI test items show that 
the tests increased in difficulty during the school year. (Note that the 
more positive difficulty estimates represent the more difficult items -) Also, 
overall, most test items fit the Rasch model and very few were deleted from 
the item bank. An average of 9h% of the items written for a particular ob- 
jective were retained. Percentages range from 75-100 for all except two ob- 
jectives, objectives 206 and 208, for which only 22% of the items had accept- 
able fit statistics. The large percentage of retained items can be attributed 
to the procedures which were used to develop the GMI test. GMI teachers pre- 
pared the test specifications and helped write and revise the test items. 
Correlations 

Table 3 contains the correlation coefficients calculated for the instruc- 
tional variables and the two test item variables. The correlations show the 
expected relationships between average item difficulty and extent of student 
mastery for Tests 1, .2 and 3, though only the correlations for Tests I and 2 
reach statistical signf icance* Thus, teachers reported that students tended 
to master objectives which had -lower average item difficulties and failed to 
master objectives assessed by the more difficult items. Although the 
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correlations for NT/M and T/NM with average item difficulty were in the ex- 
pected direction and significant for Test 3, those for the collapsed variables, 
i.e., M and NM, were not. We can speculate that differences in the distribution 
of data collected for the instructional codes, compared with Tests 1 and 2, 
may have contributed to the lower correlations. Comments made by GMI teachers 
on a supplemental questionnaire they completed at the end of the year anq. 
feedback from central office math staff suggested difficulties and problems 
regarding the probability objectives scheduled to be taught during this 
quarter. Apparently, teachers felt uncomfortable teaching these objectives 
and perhaps were unsure about student mastery. The findings for Test k were 
unusual — none of the expected correlations were found. However, a large pro- 
portion of teachers failed to complete their instructional records for many 
of the objectives. Given the relationships found for the previous tests, we 
cannot assume, at this point, that teachers did not teach and students did not 
master objectives omitted from teachers 1 instructional records. It appears 
that factors other than lack of instruction were responsible for the emissions. 
For this reason, analyses performed for variables that included "No Response," 
i.e., NR, NMNO and NTHO, cannot be interpreted. 

Correlations between the percentage of fitting items and data collected 
via teachers' instructional records failed to show the strength that those for 
item difficulty did. The correlation between the extent to which teachers 
taught an objective and the percentage of fitting items was strong and positive 
for Test 2 only. The two objectives with 22% fitting items most likely con- 
tributed to the strong correlation for that test. A much lower percentage of 
teachers taught these objectives, and of those who did, more reported non- 
mastery than mastery. 

In addition to investigating the two main hypotheses of this study, as 
second goal was to identify objectives and/or items to br^evisedj^ 
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To accomplish this, plots of the correlations between item difficulty and 
degree of student mastery and between the percentage of fitting items and 
extent of teacher instruction were drawn. These plots are illustrated in 
Figures 1 and 2, respectively. Three objectives — 326, 32? and bib — circled 
in Figure 1 were of concern since they were relatively easy objectives which 
teachers indicated they did not have time to teach. Figure 2 identifies ob- 
jectives 206 and 208 as objectives which just over half the teachers taught 
but had a small number of acceptable items. 

DISCUSSION 

The results of this study of instructional validity have beneficial 
outcomes. First, they demonstrate a true relationship between instructional 
events in the classroom and data derived from a field test designed for a 
particular course of study. Teachers 1 perceptions of the degree of student 
mastery were correlated with item difficulty. Also, if given some degree 
of variability in the quality of the items field-tested, the non- fitting 
items will tend to be those for which teachers provided less instruction. 

The second and most important benefit is that instructional data can be 
used to address pertinent questions in the test development "process, such as 
the following: Were item% that did not fit the Rasch model of poor quality, 
or was the content assessed by the item not taught and/or mastered? Should 
the curriculum or the test*, be revised? Item fit data can be employed to 
answer these questions. For example, it appears that the large proportion of 
misfitting items for objectives 206 and 208 are not due to the poor quality of 
the test items but rather to low levels of instruction and/or poor instruction 
(i.e., relatively few teachers taught these objectives; of those who did more 
had students who they classified as non-masters). These two skills measured 
similar objectives — students* ability to convert square centimeters to square 
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millimeters and cubic centimeters to cubic millimeters — and were eliminated 
from the GMI curriculum "by the math staff. In contrast, the one misfitting 
item for objective 201 (assessing metric estimation of line segments) was 
probably a poorly written item; 16% of the teachers reported that they taught 
this objective and 7h% reported student mastery. Focusing on item difficulty 
and mastery, the data led to the elimination of the three outlying objectives 
circled in Figure 1: reading and making a bar graph (326>; reading and making 
a line graph (327); solving word problems of a consumer and/or career nature 
involving coordinate graphing {hlk). It was interesting that the content of 
these three objectives was related. Since they were easy objectives charac- 
terized by low rates of instruction, . they were eliminated from the curriculum 
to allow more time for teachers to teach the more difficult objectives. 

A limitation of this study should not be overlooked. While item data were 
available for each student, only teacher level instructional and objective 
mastery data were collected. Obviously, results would be much more conclusive 
if teachers' reports of instruction and objective mastery could be collected 
at the student level and compared to each student's performance on items re- 
lated to each objective. Unfortunately, such data collection efforts are 
very time consuming for teachers, but collecting data on instructional emphases 
at the student level or teacher level should be considered for the purpose of 
test development. 

At a general' level, the study suggests a useful line of research that 
has not been explored in great detail by those involved in educational measure- 
ment. The basic design for this line of research relates instructional 
variables to the psychometric attributes of test items written to assess the 
effects of instruction. This study operational! zed instructional variables 
in terms of teacher self-reports, and other procedures for operational zing 
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instructional variables need to be explored (e.g. , direct classroom observa- 
tion). Furthermore, this study examined item difficulty and item fit for 
the one-parameter latent trait model. Future studies might explore the 
relationship between instructional variables and item discrimination, 
in both the classical and latent trait sense, and the tendency of items to 
elicit guessing* The interpretation of our psychometric analyses might be 
enhanced if we could understand whether or to what extent the psychometric 
attributes of test items are revealed in the instructional activities of 
classroom teachers. 
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Table 1-A 



Percentage of Teachers Selecting Response Options on the 



General Mathematics - 1 



Instructional Record 



\ 



' Test 1 Objectives 



Avg. 



Obj 


T/M 


NT/M 


T/NM 


NT/IT 


NT/O 


NR 


M 


NM 


UMNO 

\ 


T 


NT 


NTNO 


Diff. 




103 


66 . 


5 


8 


3 


8 


11 


71 


19 , 


30 

■ ■ .\ 


74 


16 


27 


-1.5909 


100 


105 


61 


5 


5 


15 


5 . 


8 


66 


25 


33 


,66 


25 


33 


-0;1801 


100 

1 


108 


82 


16 


, 3 


0 


0 


0 


98 


3 


3 


85 


16 


• 16 


-2.5923 


lop 


114 


68 


5 


21 


3 


0 . 


3 


73 


24 


27 


89 


.8 


11 ! 

'V 


-0 .6653 


io|o 


115 


71 


3 


13 


0 


3 


11 


74 


16 


27 


84 


6 


17 


0 .7091 


; 10 


0 




117 


71 


3 


11 


5 


0 


11 


74 


16 


27 


82 


8 


19 


-0.8167 




1° 1 


118 


53 


5 


29 


11 


0 


3 


58 


40 


43 


82 


16 


- 19 


-0.9353 


•j loo ! 


119 


66 


3 


13 


11 


0 


8 


69 


24 


32 


79 


14 


22 


-0.1993 


100 


120 


32 


8 


0 


8 


0 


3 


90 


8 


11 


82 


16 


19 


-2.8723 


89 


121 


79 


5 


5 


8 


0 


3 


84 


13 


16 


84 


13 


16 


-2.6180 


89 


122 


61 


0 


18 


13 


0 


8 


61 


31 


39 


79 


13 


21 


-1.3739 


100 


123 


68 


3 


8 


13 


3 


5 


71 


24 


29 


76 


19 


24 


-1.8790 


100 . 


125 


58 


3 


13 


18 


' 3 


5 


61 


34 


39 


71 


24 


29 


0.3001 


100 


126 


50 


0 


26 


16 


3 


5 


50. 


45 


50 


76 


19' 


24 


-0.2582 


100 


128 


42 


3 


21 ■ 


21 


3 


11 


45 


45 


56 


63 


27 


38 


-0.6099 


100 


129 


18 


0 


18 


.42 


5 


16 

i 


18 


65 


81 


36 


47 


63 


0.7376 


89 


130 


24 


0 


29 


37 


3 


8 


24 


69 


77 


53 


40 


48 


0.4576 


100 


131 


29 


0 


13 


42 


8 


8 


29 


63 


71 


42 


50 


58 


0.3909 


100 



Notes : Column headings are as follows : 
Obj = Objective; 
T/M » Taught and mastered; 

NT/M = Not taught; already mastered by students? 

T/NM "a Taught, but not mastered? 

NT/IT = Not taught? lack of instructional time? 

NT/0 = Not taught; other reason? 

NR = No response ? 

M = Mastered by students (T/M + NT/M) ? 

NM = Not mastered by students (T/NM + NT/IT + NT/O) ? 
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Table 1-A 

Test 1" (cont'd) 



Notes: NMNO » NM + NR; 

T » Taught by teachers (T/M + T/NM) ; 

NT = Not taught by teachers (NT/M +■■ NT/IT + NT/0) ; 

NTNO = NT + NR; 

Avg. "Diff. = Average difficulty of items in bank; 

% Items = Percentage of items piloted which fit the Rasch model. 
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Table 1-B 



Percentage of Teachers Selecting Response Options on the 
General Mathematics I Instructional Record 
Test 2 Objectives 



Obj 


T/M 


NT/M 


T/NM 


NT/IT 


NT/O 


NR 


M 


NM 


NMNO 


rp 


vrm 

N 1 


N IjNU 


Avg. 




201 


71 


3 


5 


11 


3 


8 


74 


19 


27 




17 


25 


-1.0771 


O ft 

89 


202 


61 


0 


18 


13 


3 


5 


61 


34 


39 


79 


16 


21 


-1.1627 


100 


203 


42 


0 


29 


16 


5 


8 


42 


50 


58 


71 


21 


29 


1 ft ^ *^ ft 

1. 0622 


10 □ 


204 


47 


0 


29 


13 


3 


8 


47 


45 


53 


76 


16 


24 


T /i ^ ft 

1.4039 


Oft 

80 


205 


68 


0 


11 


13 


3 


5 


68 


27 


32* 


79 


16 


21 


-0. 9529 


i ftft 
1UU . 


206 


24 


0 


32 


24 


8 


13 


24 


64 


77 


56 


32 


A C 

45 


0. 4022 


22 


207 


58 


0 


8 


18 


8 


8 


58 


34 


42 


66 


26 


34 


-0.0729 


rtft 

89 


208 


24 


0 


32 


26 


8 


11 


24 


66 


77 


56 


34 


A 

45 


0.7232 


22 


209 


50 


0 


29 


13 


3 


5 


50 


45 


50 


79 


16 


21 


-0.1922 


T ft ft 

100 


210 


47 


0 


32 


13 


3 


5 


47 


48 


53 


79 


16 


21 


0 .8122 


T ft 

78 


211 


50 


0 


29 


13 


3 


5 


50 


45 


50 


79 


16 


21 


-1.0605 


i ft ft 
100 


212 


37 


0 


37 


16 


3 


8 


37 


56 


64 


74 


T ft 

19 


27 


ft ft.ft ft ft 

0 • 9228 


1Q 


• 213 


32 


0 


32 


26 


3 


8 


32 


61 


69 


C A 

64 


ft Q. 

23 


37 


ft ft"7 1Q 




214 


32. 


0 


/ 37 


21 


0 


11 


32 


58 


69 


69 


21 


"3 *5 

JZ 


ft flOl "7 

□ • U82 / 


1UU 


215 


68 


' 5 


/ 

/ 16 


11 


0 


0 


73 


27 


27 


84 


16 


lb 


T Cft 1 ft 

—1. bUlU 


t nn 

1UU 


217 


71 


0 


16 


11 


0 


3 


71 


27 


30 


87 


T 1 

IX 


14 


-1. 43 J / 


i nn 

1UU 


218 


84 


0 


3 


11 


0 


3 


84 


14 


17 


87 


11 


14 


— 1 • ZOZJ 






33 


u 








XX 


55 


35' 


46 


79 


11 


22 


-0.1949 


100 


220 


47 


0 


32 


11 


0 


11 


47 


43 


54 


79 


11 


22 


0.3615 


. 100 


222 


79 


3 


3 


13 


0 


3 


82 


16 


19 


82 


16 


19 


-0.3820 


80 


223 


74 


0 


8 


13 


0 


5 


74 


21 


26 


82 


13 


18 


-0.6745 


80 


224 


74 


3 


5 


13 


0 


5 


77 


18 


23 


79 


16 


21 


-0.4895 


100 



(continued on next page) 
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Table 1-B 

Test, 2 (cont'd) 



Obj 


T/M 


NT/M 


T/NM 


NT/IT 


NT/0 


NR 


M 


MM . 


NMNO 


T 


NT 


NTNO 


Avg. 
Diff. 


% Items 


227 


61 


3 


18 


16 


0 


• 3 


64 


34 


37 


79 


19 


22 


0.2959 


100 


228 


39 


0 


32 


21 


0 


8 


39 


53 


61 


71 


21 


29 


0.6010 


89 


230 


63 


0 


16 


18 


0 


3 


63 


34 


37 


79 


18 


21 


-0.8869 


100 


233 


63 


3 


13 


16 


0 


5 


66 


29 


34 


76 


19 


24 


— □ • 04 jy 




234 


53 


3 


21 


18 


0 


5 


56 


39 


44 


74 


21 


-26 


0,0082 


100 


235 


34 


3 


26 . 


26 


0 


10 


37 


52 


62 


60 


29 


39 


0,2389 


100 



Notes : Column headings are as follows: 
Obj Objective; 
T/M * Taught and mastered; 

NT/M =* Not taught; already mastered by students; 

T/NM =» Taught , but not mastered; 

NT/IT = Not taught; lack of ins true tional time; 

NT/O = Not taught; other reason; 

NR =* No response; 

M = Mastered by students (T/M + NT/M) ; 

NM =» Not mastered by students (T/NM + NT/IT + NT/0) ; 

NMNO =» NM + NR; 

T = Taught by teachers (T/M + T/NM) ; 

NT = Not taught by teachers (NT/M + NT/IT + NT/0) ; 

NTNO = NT + NR; 

Avg. Diff. = Average difficulty of items in bank; 

% Items = Percentage of items piloted which fit the Rasch model. 



ERLC 



-16- 



19 



Table 1-C 

Percentage of Teachers Selecting Response Options on the 
General Mathematics I Instructional Record 
Test 3 Objectives 



Obj 


T/M 


NT/M 


T/NM 


NT/IT 


NT/O 


NR 


M 


NM 


NMN0 


T 


NT 


NTNO 


Avg. 
Diff. 


% Items 


302 


29 


0 


21 


24 


18 


9 


29 


63 


72 


50 


42 


51 


1.3291 


100 


305 


21 


0 


24 


24 


21 


12 


' 21 


69 


81 


45 


45 


57 


-0.0951 


100 


306 


24 


0 


26 


24 


18 


9 


24 


68 


77 


. '50 


42 


51 


1.2922 


100 


309 


74 


0 


12 


12 


0 


3 


74 


24 


27 


86 


12 


15 


-0.4591 


100 


310 


24 


.: o 


29 


32 


6 


9 


24 


67 


76 


53 


38 


47 


-0.6163 


100 


311 


32 


o 


18 


38 


6 


6 


32 


62 


68 


50 


44 


50 


-0.5709 


8.8 


312 


50 


o 


6 


35 


6 


3 


50 


47 


50 


56 


41 


44 


-0.6011 


100 


313 


35 


0 


12 


35 


9 


9 


35 


56 


65 


' 47 


44 


53 


-0.9043 


100 


314 " 


— 59 


0 


12 


15 


6 


9 


59 


33 


42 


71 


21 


30 


-1.0126 


100 


315 


50 


0 


13 


13 


5 


19 


50 


31 


50 


63 


18 


37 


-0.7807 


100 


316 


47 


0 


21 


18 ' 


6 


9 


47 


45 


54 


68 


24 


33 


0.3496 


100 


317 


47 


0 


24 


15 


6 


9 


47 


45 


54 


71 


21 


30 


-0.0169 


89 


318 


82 


0 


6 


12 


0 


0 


82 


\ 18 




88 


12 


12 


-0.1593 


89 


319 


74 


0 


12 


12 


0 


3 


74 


24 


27 


86 


12 


15 


0.3719 


100 


320 


65 


0 


18 


15 


0 


3 


65 


33 


36 


83 


15 


18 


0.3388 


100 


321 


44 


0 


38 


15 


0 


- 3 


44 


53 


56 


82 


15 


i8 


1.1108 


100 


322 


35 


0 


47 


15 


. 0. 


3 


35 


62 


65 


82 


15 


18 


0.8533 


100 


323 


29 


0 


35 


26 


3 


6 


29 


64 


70 


64 


29 


35 


0.3651 


100 


324 


47 


0 


0 


38 


3 


12 


47 


41 


53 


47 


41 


53 


0.6942 


100 


325 


47 


0 


0 


38 


■ 3 


12 


47 


41 


53 


47 


41 


53 


0.1608 


89 


326 


44 


3 


3 


38 


3 


9 


47 


. 44 


53 


47 


44 


53 


-2.1466 


100 


327 


44 


3 


3 


38 


3 


9 


47 


44 


53 


47 


44 


53 


-1.8057 


100 



(continued on next page) 
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Table 1-C 

Test 3 (cont'd) 



Ob-i 


T/M 


NT/M 


T/NM 


NT/IT 


NT/0 


NR 


M 


NM 


NMNO 


T 


NT 


NTNO 


Avg. 
Diff. 


% Items 




44 


o 


3 


38 


3 


12 


44 


44 


56 


47 


41 


53 


-1.4267 


89 


329 


15 


o 


12 


47 


9 


18 


15 


68 


86 


"27 


56 


74 


0.4803 


100 


330 


35 


o 


3 


44 


9 


9 


35 


56 


65 


38 


53 


62 


-0.6254 


100 


331 
«j «j -l 


32 


o 


6 


41 


9 


12 


32 


56 


68 


38 


50 


62 


-0.0368 


100 


332 


29 


o 


6 


44 


9 


12 


29 


59 


71 


35 


53 


65 


0.6958 


89 


333 


26 


: 0 


12 


50 


3 


9 


26 


65 


74 


38 


53 


62 


0.1655 


89 


334 


21 


0 


15 


50 


3 


12. 


21 


68 


80 


36 


53 


65 


1.6493 


75 


336 


24 


0 


6 


50 


6 


15 


24 


62 


77 


30 


" 56 


71 


-0.3895 


100 


337 


9 


0 


9 


62 


6 


15 


9 


77 


92 


18 


68 


83 


0.0004 


75 



Notes : Column headings are as follows: * 
Obj » Objective; 
T/M = Taught and mastered; 

NT/M = Not taught; already mastered by students; 

T/NM = Taught, but not mastered; 

NT/IT =» Not taught; lack of instructional time; 

NT/0 =» Not taught; other reason; 

NR = No response; 

M = Mastered by students (T/M + NT/M) ; 

NM = Not mastered by students (T/NM + NT/ IT + NT/0) ; 

NMNO =» NM + NR; 

T =* Taught by teachers (T/M + T/NM) ; 

NT » Not taught by teachers (NT/M + NT/IT + NT/0) ; 

NTNO = NT + NR; 

Avg. Diff. =* Average difficulty of items in bank; 

% Items = Percentage of items piloted which fit the Rasch model. 
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Table 1-D 



Percentage of Teachers Selecting Response Options on the 
General Mathematics I Instructional Record 
Test 4 Objectives 



Obi 


T/M 


NT/M 


T/NM 


NT/IT 


NT/0 


NR 


M 


NM 




T 






Avg. 

L/XX X • 


a X LC1UJ 


402 


38 


o 


16 


19 


o 


28 


38 




61 

VJ J 


54 


19 

X«7 


47 


0,3244 


100 


403 


38 


o 


16 


16 


0 


31 


38 


32 

•J4 


61 


54 


16 

X VJ 


47 


-0 1079 

VJ • XVJ / -s 


100 

XVJ VJ 


404 


34 


0 


19 


16 


0 


31 


34 ■ 


35 


66 


53 


16 

XV 


47 


n 2581 

VJ • l JOJ 


i nn 

XVJ VJ 


405 


28 


0 




28 


o 




28 


17 


71 

/ X 


17 




62 

VJ4 


n 5999 

U • J??? 


i nn 

XVJ VJ 


406 


25 


0 


13 


28 


0 


34 


25 


41 

*rX 


75 


18 


28 


62 

VJ tm 


n 5n45 

VJ • JU4 J 


i nn 

XVJ VJ 


407 


25 


o 


13 


28 


3 


31 

J X 


25 


44 


• 75 


18 


11 
J l 


62 

VJ4 


-0 2416 


i nn 

XVJ VJ 


411 

** XX 


22 


o 


VJ 


41 

4X 


n 

VJ 


11 

•J X 


22 

£4 


47 

4 / 


78 


28 

4 O 


41 

41 


72 


-n 9fi5Q 
U • 4U37 


PQ 

07 


412 


16 


n 
\j 


q 


41 

4X 


n 


j4 


1 6 

X VJ 


— / VJ 


P4 

04 


25 

4 -J 


41 

4 X 


75 


n 4nQ4 

U • 4U74 


i nn 

1UU 


41 1 


1 1 

X -J 


n 
vj 


* 17 


41 

4 1 


n 

VJ 


14 
J4 


1 1 

1 J 


54 


PP 


9fi 
• & o 


41 

41 


75 


n fiiin 


PQ 
07 


41 4 
414 


c 
o 


u 


1 Q 


41 
41 


1 i 


11 
J 1 


fi 


fil 

D J 


Q4 
74 


95 


44 
44 


75 


—1 ^HfiQ 
~1 • J307 


i nn 

1UU 


41 5 


51 


n 

u 


Q 


1 fi 
lv3 


n 


9 9 


51 


43 


47 
4 / 


fi9 

04 


1 fi 

io 


IP 


n 4in5 

U • 4jUj 


.1 nn 

1UU 


41 £ 


47 

4 / 


n 
u 


1 7 
1 J 


1 Q 
17 


n 

u 


00 


47 
4 / 


1*5 


54 
34 


fin 


1 Q 
17 


41 
41 


n 71 7P 

U m /I/O 


i nn 

1UU 


41 7 

41 / 


5n 


n 

u 


Q 
7 


1 Q 
17 


n 

u 


44 


. 5n 

* 3 U 


9P 


5n 

JU 


5Q 

3 7 


1 9 

17 


41 

41 


n nP7fi 
u • ua / o 


i nn 

1UU 


41 P 
410 


44 

44 


n 

vj 


1 fi 
lO 


1 Q 
17 


n 

u 




44 
44 


15 
jj 


57 

3 1 


fin 

ou 


1 Q 

17 


41 

41 


1 51QP 


PQ 
07 


41 Q 

417 


1A 
j4 


n 
u 


1 fi 
ID 


9P 


n 
u 


22 

£4 


14 
J 4 


AA 
44 


fifi 
OO 


5n 


9 G 


5n 


1 « 3740 


PQ^ 
07 


420 

** ^ VJ 


25 

£ 3 


n 
vj 


Q 

7 


44 

44 


VJ 


22 


25 


51 


75 


14 


44 

44 


b6 

VJ VJ 


h 11P4 


i nn 

1UU 


424 
4^4 


1 Q 
17 


n 
vj 


fi 
vj 


44 

44 


n 

VJ 


11 

-J X 


l^ 
1^ 


5n 


81 

Ol 


25 


44 

44 


75 


i n52 1 


i nn 

XVJ VJ 


425 


13 


o 


9 


47 


0 


31 


13 


56 


87 


22 


47 


78 


1. 3881 


100 


426 


9 


0 


13 


47 


0 


31 


9 


60 


91 


22 


47 


78 


1.0659 


100 


427 


9 


0 


13 


47 


0 


31 


9 


60 


91 


22 


47 


78 


1.1811 


100 


428 


16 


0 


9 


50 


0 


25 


16 


59 


84 


25 


50 


75 


0.8726 


100 


430 


22 


0 


9 


41 


0 


28 


22 


50 


78 


31 


41 


69 


0.5518 


100 



(continued on next page) 
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Table 1-D 
' -Test* 4 (cont'd! 



Notes ; Column headings are as follows: 
Cbj = Objective; 
T/M =« Taught and mastered; 

NT/M = Not taught; already mastered by students; 

T/NM = Taught/ but not mastered; 

NT/IT = Not taught; lack of instructional time; 

NT/0 » Not taught; other reason; 

NR = No response; 

M * Mastered by students (T/M + NT/M) ; 

NM = Not mastered by students (T/NM + NT/IT + NT/0) ; 

NMNO + NM + NR; 

T a Taught by teachers (T/M + T/NM) ; 

NT = Not taught by teachers (NT/M +-<flT/lT + NT/0) ; 

NTNO ^ NT + NR; \ 

Avg. Diff. = Average difficulty of iteqis in bank; 

% Items = Percentage of items piloted which fit the Rasch model. 
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Table 2 

Descriptive Statistics for Variables Used in Analyses 





TUST 1 
(18 objoctlves) 


TEST 2 
(16 objactivia) 


TEST 2 
(31 objectlvea) 


TEST 4 
(22 objectlvai) 


TESTS 1-4 
(99 objectives) 


Variable 




S.P. 


Hanga 

►!lp/HH 


Mean 


..ii&t 


Range 

til _ /U kM 




. £■!>■ 


Ranqa 

HLn/HnX 


.!iS»!i „. 


SJJa 


Range 


neai) 


£ n 

D*V, 




Taught ant] metered (1) 


SB.) 


19.1 


18/82 


51.9 


16.8 


24/B4 


irt ft 

19.9 


11 1 

17.7 


9/d2 


Ml C 

«0 ,0 


h q 






2n i 

4U» J 


6/84 


Hot taucjlitj Already 
nan tend (2) 


1.1 


1.8 


0/16 


0.9 


1.5 


0/5 


0.2 


0.7 


0/3 


0 


Q 


- 


1.0 


2.3 


0/16 


Taught | not mastered 1 3) 


14.1 


8.7 


0/29 


21.2 


11.0 


1/17 


14.6 


11 at 


n/Jl 




1 4 
J i ? 




15 B 


All i • 


0/47 


Hot Uughtj lack of 


14.7 


n.2 


0/42 


15.9 


4.9 


11/26 


10.9 


14.2 


12/62 


12.7 


12.1 


16/50 


24.1 


14.1 


0/62 


time (4) 






























* — 


Nut tauijlit j otliur 


2.4 


2.7 


0/8 


2.0 


2.6 


0/8 


5.8 


5.1 


0/21 


0.1 


0.9 


0/1 


2.9 


4.0 


0/21 


reason (5) 






























■ — 


No reujionae (6) 


7.1 


4.0 


0/16 


6.5 


3.2 


0/11 


9.0 


4.6 


0/19 


28.5 


4.6 


22/14 


12.1 


9.7 


0/34 


Mastered by students 


62.0 


21.9 


18/98 


54.8 


17.4 


24/84 


40,1 


17.7 


9/82 


26.6 


13.9 


6/51 


45.1 


21.6 


6/911 


1112) 
































Hot waotcrftd by 


21.] 


19.6 


3/69 


19.1 


15.1 


14/66 


51.1 


15.5 


18/77 


45.0 


11.4 


25/61 


42.8 


16.6 


3/77 


atudents (3Ml5) i 
































Not watered by 


36.4 


21.6 


1/81 


45.6 


17.4 


17/77 


60.1 


17.8 


18/92 


71.5 


11.8 


47/94 


55.1 


21.5 


3/94 


studentB plua (6) 
































Taught by teacher (1*3) 


72.4 


15.1 


16/89 


75.0 


8.1 


56/67 


54.5 


19.2 


18/86 


18.6 


14.8 


22/62 


60.0 


20.6 


18/69 


Hut 1 Atinlif liu t«fl/'iiiif 


20.9 


12.8 


6/50 


18.8 


6.2 


11/14 


16.9 


16.0 


12/68 


31.0 


12.4 


16/50 


28.0 


14.6 


6/60 


(2I4ISI 
































Nut taught by ttacbor 


28.0 


14.8 


11/61 


25.4 


8.3 


14/45 


45.9 


19.2 


12/82 


61.5 


14.6 


18/76 


40.1 


20.5. 


11/83 


plus (6) 
































Item difficulty 


■0.7776 


1,1603 


-2.B721/ 
0.7176 


-0.1610 


0.8100 


"1.6010/ 
1.4019 


•0.0577 


0.8942 


-2.1466/ 
1.6491 


0.5261 


0.6766 


-1.1569/ 
1.5948 


•0.0881 


0.9649 


-2.6721/ 
1.6493 


r«rceiita<je of fittiny 


98.2 


4.2 


89/100 


87.3 


20.9 


22/100 


95.9 


7.1 


75/100 


98.0 


4.1 


89/100 


94.3 


12.8 


22/lim 


Hows 
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Table 3 



Correlation Coefficients for GMI Quarters 1 - 4 



instructional 
Code 


21 


Item Difficulty 
Q2 03 04 


Q1-Q4 


21 


Percentage of Pitting Items 
Q2 03 04 


21:24 


T/M 


-.71*** 


-.70*** 


-.23 


.04 


-.55*** 


-.03 


.48** 


.25 


-.06 


.11 


NT/M 


-.66** 


-.24 


-.57*** 


0 


-.55*** 


-.07 


.21 


.15 


0 


.09 


T/NM 


.56* 


.65*** 


.46** 


-.17 


.36*** 


.34 


-,26 


.19 


-.09 


-.19 


NT/IT 


.53* 


.45* 


-.09 


.14 


.33*** 


-.16 


-.54** 


-.47** 


.02 


-.06 


NT/0 


.44 


.33 


.17 


-.63** 


.08 


.13 


-.64*** 


.15 


.15 


-.13 


M ' 


-.74*** 


-.70*** 


-.25 


.04 


-.58*** 


-.04 


.48** 


.25' 


-.06 


.11 


NH 


.69** 


.68*** 


.32 


.04 


.52*** 


.06 


-.47* 


-.24 


0 


-.20* 


NMNO 


.74*** 


.70*** 


.26 


-.04 


.58*** 


.05 


-.49** 


-.25 


.05 


-.12 


T 


-.58* 


-.56** 


.06 


-.01 


-.37*** 


.15 


.63*** 


. .34 


-.08 


.01 


NT 


.47* 


.44* 


-.05 


.09 


.26** 


-.16 


-.65*** 


-.36** 


.03 


-.08 


NTNO 


.58* 


.56** 


-.06 


,01 


.37*** 


-.15 


-.66*** 


-.35 


.07 


-.02 



*p< *05 
**p< .01 
***p< .001 



Figure 1 

Plot of Student Mastery and Item Difficulty 
(Symbol is Test) 
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•/ Figure 2 

Plot of Instructional Emphasis and Percentage of Fitting Items 



206 S 208 



0 



34 

3 



3 



4 4 -4 • 



+ -4 4- * + "I « i + " • 



1 

3 
1 

2 

4 

3 
4 
1 

3 



3 
1 

3 
4 
4 



1 

0 



~3" 
O 



~4 
O 



6 
0 



7 
0 



U 
O 



• 4 -4 - * 

1 

V V O 
0 ii 0 



Percentage of Fitting Items 

2.9 



ERIC 



APPENDIX A 



Ninth Grade General Mathematics I Objectives 
Assessed by the GMI Area Exam 

Level 1 

Code Objective: "The student will be able to, . . " 

103 solve equations in one variable using addition and subtrac- 

■ tion. m 

105 read and write whole numbers and standard numerals for 

powers of 10 and vice versa. 

108 multiply whole numbers with a 2-digit multiplier. 

114 divide whole numbers with a 3-digit divisor. 

115 use order of operations for computation involving addition/ 
subtraction, multiplication and division. 

117 identify the place value of a decimal through the millionth 
place. 

118 read and write the word name of a mixed decimal fraction 
or vice versa. 

119 compare decimals by selecting the correct inequality or 
equality sign that indicates their order. 

120 add decimals. 

121 subtract decimals. 

122 solve word problems of a consumer and/or career nature 
involving addition and subtraction of decimals. 

123 multiply decimals. 

125 mu±^i?iy~6r-Hto JulQ .20- JLmQying , 

the decimal point to the left or right). 

126 divide a decimal by a decimal with a one, two or three 

place decimal in the divisor. 

128 round decimal quotients to the nearest tenth, hundredth, 
or thousandth. 

129 write a standard numeral for a number expressed in scien- 
tific notation. 

130 solve word problems of a consumer and/or career nature 
involving multiplication and division of decimals.- 

131 solve addition and subtraction problems involving 
different times. 
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Ninth Grade- General Mathematics I Objectives 
Assessed by the GMI Area Exam 

Level 2 

Code Objective; The student will be able to. . ." 

201 estimate and measure the length of a segment to the 
nearest centimeter and/or millimeter. 

202 select the appropriate linear measure using the metric 
units - kilometers , meters and centimeters . 

203 Add two measurements expressed in yards and inches or in 
feet when given a problem requiring regrouping. 

204 make conversions of linear measure using the metric units 
kilometers, meters, and centimeters. 

205 find the area of figures by counting square centimeters. 

206 convert square centimeters into square millimeters. 

207 find the volume of figures by counting cubic centimeters. 

208 convert cubic centimeters to cubic millimeters. . „ 

209 select the appropriate metric measure to measure the 
capacity of a given container ([liter, milliliter). 

210 make conversions from liter to milliliter and vice versa. 

211 select the appropriate measure to measure the weight of 
a given. object (metric or standard) . 

212 make conversions between the following metric units of 
mass: kilogram, gram, milligram. 

213 when given labeled pictures of thermometers in Celsius, 
select the appropriate: metric unit of temperatures from 
descriptive phrases dealing with temperatures such as ~ 
warm bread, snqwball, etc. 

' 214 solve word problems of consumer and/or career nature 

dealing with metric units of measure (length, mass_, 

capacity and temperature). 

215 write a fraction or a mixed number for shaded regions 

and/or number lines . \ 

217 find equal ratios by multiplication. 

213 change improper fractions to mixed numbers or whole 

numbers and vice versa. 

219 write a fraction as a terminating decimal . 
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u Ninth Grade General Mathematics I Objectives 
Assessed by the GMI Area Exam 

Level 2 (continued) 

Code Objectives; The student will be able to, . . " 

220 write a faction as a nonterminating decimal and round to 

the nearest thousandth. 

222 multiply a proper, fraction by a whole. ' number . 

223 multiply three or more proper fractions. 

224 Write the reciprocal of a proper fraction or mixed number. 

227 multiply and/or divide a mixed number by a mixed number. 

228 solve word problems of a consumer and/or career nature in- 
volving multiplication and division of fractions and mixed 

, numbers. 

230 find the lowest common denominator of two or three mixed 

numbers. 

233 add two mixed numbers with like or unlike denominators. 

234 subtract two mixed numbers with like or unlike demoninators 
and regroup. 

235 solve word problems of a consumer and/or career nature 
involving addition and subtraction of fractions or mixed 
numbers . 



Level. 3 

Code Objective:- "The student will be able to. . . " 
-302 find the -pr ob abi rl-i-t-i-es— using— tree - di agrams-. 

305 use experiments to make predictions. 

306 solve word problems involving probability. 



309 find the missing number in a proportion by using cross pro- 
ducts . 

310 interpret and use scale drawings and maps, when the scale 
factor has been given, to determine appropriate distance. 

311 identify pairs of congruent segments or polygons, recog- 
nize a pair of lines which are parallel or a pair which 
are perpendicular and construct a perpendicular bisector 
of a segment. 

. 312 identify acute, right, obtuse and congruent angles . 
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Ninth Grade General Mathematics I Objectives 
Assessed by the GMI Area Exam 

Level. 3 (continued) 



Code Objective: "The student will be able to, > > " 

313 identify triangle, qualrilateral , pentagon, hexagon, 
octagon, prism, cube, pyramid, sphere, cylinder and cones. 

314 identify similar figures. 

315 identify corresponding angles and sides of similar figures 

316 find missing dimensions in similar triangles by using 
proportion. 

317 solve word problems involving ratio, proportion and simi- 
larity. 

318 write percents as decimals and vice versa. 

319 ~~v?rite percents as fractions and vice versa. 

320 find a percent of a number. 

321 find what percent one number is of another. 

322 find a number when a percent of it is known. • 

323 solve word problems of a consumer and/or career nature 
involving percent (include interest problems). 

324 find the. mean of a set of numbers. 

325 find the median and the mode of a set of numbers. 

326 read and make a bar graph. 

327 read and make a line graph. 

328 read and make a, circle graph. 

329 solve word problems of a consumer and/or career natu re 
involving statistics (mean, median, mode, graphs) . 

330 plot positive and negative numbers on a number line. 

331 select the proper inequality or equality sign to compare 
positive and negative numbers. 

332 arrange a set of integers in order from least to greatest. 

333 add positive and negative numbers. 

334 subtract positive and negative numbers. 
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Ninth Grade General Mathematics I Objectives 
Assessed by the GMI Area Exam 

Level. 3 (continued) 



Code Objective: "The student will be able to, . . " 

336 divide positive and negative numbers. 

337 solve word problems of a consumer and/or career nature 
involving positive and negative numbers. 



Level 4 

Code Objective: "The student will be able to. . . " 

4-02 evaluate expressions involving order of operations, 

(involving variables) 

403 solve addition and subtraction equations involving integers 
and decimals. 

404 solve multiplication and division equations involving 
integers and decimals. 

4 05 solve two-step equations. 

40 6 combine like terms to solve equations. 

4 07 solve word problems of a consumer and/or career nature 

involving equations. 

411 give ordered pairs of integers for points in. four 
quadrants and locate points for ordered pairs of integers. 

412 read the graph of an equation on a four quadrant grid to 
complete a table of ordered pairs. 

413 draw the grapn of~a7~given equation on a fotor quadrant grid. 

414 solve word problems of a consumer ^ and/or career nature 
involving coordinate graphing. 

415 find the perimeter of a geometric figure. 

416 find the circumference of a circle. 

417 find the area of a rectangle, a square and a pa -^llelogram. 

418 find the area of a triangle and a trapezoid. 

419 find the area of a circle, or find the radius and give the 
area. 




Ninth Grade General Mathematics I Objectives 
Assessed by the GMI Area Exam 



Level 4 


(continued) 


Code 


Objective: ' "The student will be able to. • 


420 


solve word problems of a consumer and/or career nature 
involving perimeter and area. 


424 


find the volume of a rectangular prism and a cube. 


425 


find the volume of a cylinder. 


426 


find the volume of a pyramid and a cone. 


4.27 


find the volume of a sphere. 


428. 


solve word problems of a consumer and/or career nature 
involving surface area and volume- 


430 


measures angles by using a protractor. 
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